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ABSTRACT 



This two-year study involving five colleges and universities 
compared the academic achievement, as measured by the Educational Testing 
Service (ETS) Major Field Achievement Test (MFAT) in Business of students in 
traditional undergraduate programs and those in non-traditional accelerated 
adult degree programs. The study also compared the subjects' test results 
with national norms and analyzed the relationship of the MFAT and grades in 
traditional and non-traditional business programs. In Phase 1, 122 
traditional undergraduate students and 209 non-traditional students took the 
pre- assessment ETS MFAT and 81 students completed the post -assessment MFAT. 
Undergraduate non-traditional students averaged slightly higher on the ETS 
MFAT pre -assessment than did traditional undergraduate students. During Phase 
2, 173 of the students who took the ETS test originally re-took it after 
completing their major courses. This study compared matched pairs of 
traditional/non-traditional students and also found that non-traditional 
students scored higher on the post -assessment than did traditional students. 
Results indicate that students in non-traditional accelerated business 
programs score as well, if not better, than traditional students in business 
programs at the same institutions. (Contains 19 references.) (DB) 
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Non-traditional vs. Traditional Academic Delivery Systems: Comparing ETS 
Scores for Undergraduate Students in Business Programs, 1996-1999 

Abstract 

Six different colleges and universities from across the United States participated in a two- 
year assessment research project, comprised of two Phases, comparing the academic achievement of 
students in similar traditional and non-traditional undergraduate programs in Business. The main 
goals of the research project were to a) compare and contrast the academic achievement in 
traditional and non-traditional Business programs through pre- and post-assessment by using the 
Educational Testing Ser\dce (ETS) Major Field Achievement Test (MFAT) in Business, b) compare 
the data with national norms, and c) analyze the relationship of the MFAT and grades in traditional 
and non-traditional Business programs. 
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Non-traditional vs. Traditional Academic Delivery Systems: Comparing ETS 
Scores for Undergraduate Students in Business Programs, 1996-1999 

Introduction 

The higher educational community is currently engaged in a dichotomy of two major trends, 
one centered on the increased emphasis on traditional beliefs in assessment (modes of instruction) 
and the second is a growing number of non-traditional programs for career-oriented adults. 
Traditional accrediting bodies are mandating assessment as an integral component of the 
accreditation process. Regional accreditation associations typically focus on establishing 
assessment guidelines for determining student academic achievement and institutional effectiveness 
of traditional curricula and delivery systems. However, non-traditional programs and delivery 
systems are unique and do not always fit into typical assessment models. While traditional regional 
accrediting bodies are focusing on assessment, more and more institutions are offering non- 
traditional programs that include distinctive delivery systems. Correspondingly, non-traditional 
adult degree programs are being developed at an accelerated rate. Enrollment in these non- 
traditional programs increases at a steady rate, due mainly to the major shift in the college clientele. 
Today, the majority of college students attends school part-time or in the evening, is employed full- 
time, and/or commutes to campus. They are a diverse group of students that spend less time on 
campus. 

This rapid growth of adult degree programs has raised questions in some areas of academe 
concerning the effectiveness of non-traditional programs compared to the more traditional delivery 
systems. This research project is designed to address these two entities by completing a two-year 
assessment study of student academic achievement for both non-traditional and traditional Business 
programs. "Non-traditional programs,” for the purposes of this paper, are defined as degree 
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programs for the working adult student returning to college. These programs have accelerated 
delivery systems, curriculum, and pedagogy designed for the working adult student returning to 
school to complete his/her degree. They are typically scheduled for evenings or weekends, and 
students focus on one course at a time, completing it. then proceeding to the next in a cohort model. 

Nature of Research Questions 

While every institution of higher education needs clear and publicly stated purposes, 
consistent with their mission, there is a growing demand for institutions to verify the 
accomplishments of their education and other purposes, e.g. assessment or institutional 
effectiveness (Commission on Institutions of Higher Education, 1995). This requires an institution 
to provide evidence of its effectiveness in accomplishing its institutional purposes, specifically 
student academic achievement, student development, program quality, and institutional climate. 

In order to address this concern, twelve colleges and universities associated with the 
Consortium for the Advancement of Adults in Higher Education (CAAHE) with similar curriculum 
in non-traditional and/or traditional Business programs agreed to conduct a joint assessment project 
to compare and contrast the student academic achievement in their respective programs and 
traditional Business programs. The intent of the study was to compare outcomes of non-traditional 
and traditional delivery systems, curriculum, and student academic achievement. 

Representatives (faculty and staff) from these institutions reviewed the outcomes of their 
respective programs to determine if the ETS Major Field Achievement Test in Business matched the 
program outcomes. After a careful analysis, six of the schools determined they offered traditional or 
non-traditional Business programs with academic outcomes related directly to the ETS MFAT. 
Purpose of Study 

The central points of this research were to develop an assessment project for non-traditional 
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Business programs that addresses the guidelines of traditional regional accrediting bodies while 
comparing and contrasting the academic achievement of non-traditional and traditional students in 
the two diverse delivery systems. The three main goals of the research project were to a) compare 
and contrast the academic achievement in traditional and non-traditional undergraduate Business 
programs at a select set of colleges and universities through pre- and post-assessment by using the 
Educational Testing Service (ETS) Major Field Achievement Test (MFAT) in Business; b) compare 
the ETS results from the students involved in the research project with national norms for the 
standardized, norm-referenced test, and c) analyze the relationship of the MFAT and grades in 
traditional and non-traditional Business programs. 

The hypotheses are: 1) there is no significant difference between post-test scores of 
traditional vs. non-traditional students completing the ETS MFAT; 2) there is no significant 
difference between pre- and post-scores of traditional vs. non-traditional students; 3) there is no 
significant difference between the mean scores of traditional and non-traditional students compared 
to the national mean of the ETS MFAT; and 4) there is no significant correlation between the pre- 
and post-test scores and the grade point averages corresponding to the time the test was taken. In 
each case a .05 significance level was used. 

Project Development 

Representatives from the Consortium for the Advancement of Adults in Higher Education 
(CAAHE), with support from the Institute for Professional Development (IPD), initiated and 
developed the research questions and research project outline. Representatives from the eleven 
CAAHE institutions met during August 1 995 in Chicago. IL to develop the assessment process and 
research project. 

Data analysis was to include comparisons of test results between the two groups of students. 
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and national norms, as established by ETS. The research group selected the ETS MEAT because of 
its validity and reliability, as well as its national reputation in academe as a norm-referenced test. A 
statistical comparison was used to compare the aggregate academic achievement of both sets of 
students. The grade point averages of students was collected to determine their effect on the test 
results. 

Assessment Philosophy 

After the goals and objectives of the research project were defined, the research group 
examined both summative and formative assessment instruments in the initial stages of 
development. After careful analysis, the research group selected the ETS MEAT, a summative 
evaluation instrument that is norm-referenced. The research group also established guidelines to 
monitor the quantity of data accumulated. The group developed measures and guidelines to ensure 
the validity of data collection and that all students take the tests seriously. In addition, the 
coordination between the schools, traditional and non-traditional programs, staff, and students 
presented various challenges. The distance between all of the schools and programs, 
communication, and coordination of standardized procedures magnified these concerns. 

Limitations 

In order to address the concerns of academic integrity and validity, the research group opted 
to use the ETS Major field Achievement Test in Business. The cost of the test limited somewhat 
participation, but a grant from the Institute for Professional Development assisted participating 
schools. In addition, participants wanted to ensure compatibility of the assessment instruments with 
the outcomes of the different programs and specific courses. All of the institutions utilized the 
personnel at their respective institutions to ensure that the test measured the designated outcomes of 
the Business majors and specific courses. The plans for the research project were finalized at a 
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meeting in Phoenix during June 1996. 

However, because this research project extended beyond two years, personnel at many of the 
institutions changed, thereby interrupting some of the flow of information. Schools administered 
the test in varying fashions, e.g. part of specific courses, volunteers, graduations requirement, 
thereby creating an inconsistent environment for student participation. In addition, a number of 
students did not successfully complete their academic majors, withdrew, or transferred to another 
institution. Therefore, a total of 173 out of 333 (51 .9%) completed both the pre- and post- 
assessment instruments. 

Procedures: Data Collection 

The research project consisted of two main phases, the first phase implemented in 1996- 
1997 and the second in 1998-1999. Throughout the 1996-1997 academic year, six CAAHE schools 
administered the ETS Major Field Achievement Test as a pre-assessment instrument to both 
traditional and non-traditional undergraduate students in similar undergraduate Business programs. 
In addition, several schools actively use the ETS Major Field Achievement Test as part of their 
regular assessment programs, so the schools administered the test as a post-assessment to a 
sampling of the students. However, it must be noted that the scores of students completing the post- 
assessment test in 1996-1997 were only included in Phase I of the research project, did not complete 
the test as a pre-assessment, and thus are not matched-pair set of records. 

The second phase of the research project focused on the test results of matched-pairs of 
students, e.g. the same students completing both the ETS MFAT as both pre- and post-assessment. 
Individual schools were responsible for administration of the tests. Participating schools 
administered the ETS MFAT in 1998-1999 as a post-assessment instrument to as many students as 
possible that completed the same test as a pre-assessment in 1996-1997. The test results of these 




matched-pair of students proved to be one of the focal points of the research. In addition, the grade 
point averages of these students prior to entry and upon completion of their respective programs 
were included in the assessment research. 

Test Results from Phase I 

In Phase I of the research project, 122 traditional undergraduate students and 209 non- 
traditional students participated in the pre-assessment component of the research project. For each 
of the schools participating in the research, Table 1 lists the number of traditional and non- 
traditional undergraduate students completing either the pre-assessment or post-assessment, along 
with the average ETS raw score and standard deviation for the group. Because School A did not 
have any students participate in the pre-assessment, it would not be included in Phase II of the 
project, so the total number of schools decreased from six to five. 

Table 1: Summary of Phase I Results 



Pre-assessment: Traditional Students 





School 


School 


School 


School 


School 


School 




A 


B 


C 


D 


E 


F 


:avg 


NA 


141.4 


136.8 


144.2 


145.5 


NA 


,STD 


NA 


12.2 


8.4 


10.6 


10.9 


NA 


,N 


NA 


29 


34 


33 


28 


NA 






Pre-assessment: Non-traditional Students 






School 


School 


School 


School 


School 


School 




A 


B 


C 


D 


E 


F 


AVG 


146.5 


148.4 


145.1 


138.9 


146.9 


NA 


STD 


1 1 


7.8 


12.4 


8.4 


10.4 


NA 


N 


46 


55 


44 


12 


52 


NA 






Post-Assessment 








Traditional Students 




Non-traditional Students 






School D 


School F 




School E School F 




,AVG 


151.2 


154.5 


162.1 


152.5 




iSTD 


14.4 


14.6 


6.9 


^ 15.4 




:N 


14 


28 


19 


20 





Tables 2 to 4 list the overall results for all of the undergraduate students participating in the 
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research project. As noted in Table 2, a total of 333 traditional and non-traditional students 
completed the pre-assessment ETS Major Field Test (MFAT) with a 144.7 average and 10.8 
standard deviation. Eighty-one traditional and non-traditional undergraduate students completed the 
post-assessment MFAT with a 155.2 average, and a standard deviation of 13.7. This compares to a 
national mean of 155.6, with a standard deviation of 13.8. The MFAT is used only as a post- 
assessment instrument, therefore there are no national comparison data for the pre-assessment 
component. 

Table 2: Totals for both Traditional and Non-traditiona Students 



Pre-assessment Post-assessment National 

ETS MFAT ETS MFAT Mean 



AVG 


144.7 


155.2 


155.6 


STD 


10.8 


13.7 


13.8 


N 


333 


81 


44,686 



Table 3 lists the data for traditional students completing the MFAT as both post-assessment 
and pre-assessment. Table 4 lists the same data for non-traditional students. Non-traditional 
undergraduate students participating in the research project achieved a slightly higher average on the 
MFAT than the traditional students (146.3 vs. 142.0) did as part of the pre-assessment testing. The 
standard deviation for each group was 1 0.4 and 1 1 .0 respectively, which is lower than the national 
mean of 13.8. Non-traditional students also scored higher than both traditional students and the 
national mean on the ETS post-assessment (157.2 vs 153.4 and 155.6). This same trend will be 
evident in Phase 11 of the research project. 
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Table 3: ETS Data for Traditional Students 



National 





Pre-assessment 


Post-assessment 


Mean (post only) 


Mean 


142.0 


153.4 


155.6 


STD 


1 1.0 


14.4 


13.8 


N 


124 


42 


44,686 


Table 4: ETS Data for Non-traditional Students 



National 





Pre-assessment 


Post-assessment 


Mean (post only) 


Mean 


146.3 


157.2 


155.6 


STD 


10.4 


12.9 


13.8 


N 


209 


39 


44,686 



Chart 1: ETS Test Results of Traditional vs Non-traditional Students in Phase I (1996-97) 



ETS Major Field Test 




Demographic Data 

Because of a focused attention on academic assessment, only a limited amount of basic 
demographic data were collected on students in the research project. An attempt was made to 
collect retention data, but because of the change of personnel at participating schools, a number of 
students did not complete the post-assessment test or were not tracked for a variety of reasons. 
Therefore, these numbers did not prove useful to the study. Of the 4 1 3 students involved in the 
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study 2 1 8 were male and 1 95 were female. One hundred, seventy-three students completed both the 
pre- and post-assessment tests and 105 students (60.7%) of these students were male, while 68 
(39.3%) were female. 

Conclusions from Phase I 

During Phase I of the study, 124 traditional students and 209 non-traditional undergraduate 
students completed the ETS MFAT as part of the pre-assessment. These same undergraduate 
students were selected to complete the MFAT after completion of the courses in their major. 

During Phase I, non-traditional students in the research project averaged slightly higher on 
the ETS MFAT pre-assessment (146.3) than traditional undergraduate students (142.0). Using a 
sample group of students as a pilot for the post-assessment, undergraduate non-traditional students 
also averaged slightly higher on the ETS MFAT post -assessment (157.2) than traditional 
undergraduate students (153.4). The national mean is 155.6 for 1995-1996. 

Conclusions from Phase II 

Phase II of the research project extended into 1998-99 with 173/333 or 51 .9% of the students 
taking the pre-assessment test in 1996-1997 completing the same post-assessment test. During 
Phase II, the research focused on matched-pairs of data, totals, and select group of subtotals. 
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Table 5 : Data for All Matched-Pair Students Involved in Assessment Project 
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Table 6 and 7 are directly related to the research hypotheses. These tables depict the 
information for matched-pairs of students, separating traditional and non-traditional students. Table 
6 indicates that 46 traditional students from three different institutions completed both the ETS 
MFAT test for both the pre-assessment and post-assessment. The mean ETS score increased from 
141.8 (9.8 STD) to 150.1 (1 1.4 STD) for the post-assessment results. Table 7 indicates that 127 
non-traditional students from four different institutions scored an average of 146.1 (10.4 STD) on 
the ETS MFAT for the pre-assessment and 157.3 (1 1.7 STD) on the post-assessment. Therefore, 
non-traditional students tended to score higher on the ETS pre-assessment than traditional students 
(1 46. 1 vs 141 .8). Matched-pairs of non-traditional students also scored higher than traditional 
students on the ETS post-assessment (1 57.3 vs 1 50. 1 ). The mean of the total group of non- 
traditional students increased their scores by 1 1 .2 points while the mean of the total group of 
traditional students increased by 8.3 points. 

Chart 2 provides a graphic depiction of a comparison of the matched-pair set of traditional 
and non-traditional undergraduate students completing the ETS MFAT as both the pre-assessment 



and post-assessment. 




Table 8: Comparison Data for Matched-Pair, Traditional 
and Non-traditional Students, of Total Group 



Data 


Traditional 

Students 


Non-traditional 

Students 


National 
Mean; 1996 


Pre-ETS 

N 


46 


127 


44,686 


STD 


9.8 


10.4 


13.8 


Score 


141.8 


144.5 


155.6 


Data 


Traditional 


Non-traditional 


National 




Students 


Students 


Mean; 1998 


Post-ETS 

N 


46 


127.0 


10,830 


STD 


11.4 


11.7 


13.8 


Score 


150.1 


154.7 


154.8 



Pre-GPA 

N 


35 


115 


GPA 


2.79 


2.88 


Post-GPA 


N 


36 


115 


GPA 


3.35 


3.47 



Table 9 attempts to summarize all of the data for the entire research project. As noted in the 
table, whether using matched-pairs or total populations, non-traditional students scored higher on 
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both the ETS pre-assessment and post-assessment than traditional students. Moreover, as a group, 
both as matched-pair and total populations, non-traditional students scored higher than traditional 
students and the associated national mean. There are many possible reasons for these results, 
including the processes for administering the assessment instruments. Institutions typically made 
the ETS MEAT test part of an introductory or senior seminar/capstone course for non-traditional 
students. While several institutions did not follow the same format with traditional students, instead 
opting to have traditional students volunteer to take the assessment. Therefore, these students may 




not have taken the test as seriously as non-traditional students. 

Experts typically do not consider grade point averages a formal measure of assessment. 
Therefore, these ordinal data were collected to check the validity of the study and provide another 
avenue of research not detailed assessment. Chart 3 provides an overview of the changes in GPA 
for both traditional and non-traditional students. 

Nevertheless, the data indicate that students in non-traditional accelerated Business 
programs score as well, if not better than, traditional students in Business programs at the same 
institutions and across the nation as compared to the mean ETS scores. Moreover, when comparing 
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matched-pairs of students, the total population of non-traditional students increased its grade point 
average by .59 (2.88 to 3.47) as compared to .46 (2.79 to 3.35) for traditional students from entry 
into the undergraduate program to completion. 

Report and Conclusions Regarding the ETS/MFAT 

1) Hypothesis: there is no significant difference between post-test scores of traditional vs. non- 
traditional students . 

This hypothesis is rejected. Comparing post-test scores of these groups of students resulted 
in a mean difference of 7.17 (t = 3.581, p = .000). In other words, non-traditional students scored 
significantly higher than traditional students did on this test. The variances of the scores of the two 
groups are not significantly difference (F = .002, p = .960). 

This finding is consistent between schools for which complete data are available. The 
significantly higher scores of non-traditional students on this test is encouraging for those who 
encourage newer, innovative academic programs, and must address the concerns of quality of more 
traditionally minded practitioners. 

2) Hypothesis: there is no significant difference between pre- and post-scores . 

This hypothesis is rejected. Comparing matched pairs of pre- and post-test scores show a 
significant increase in scores for all students (t = 12.98; p = .000). This finding is robust when the 
results are disaggregated to compare pre- and post-test scores for traditional vs. non-traditional 
students sorted by college. In other words, traditional and non-traditional students in all colleges in 
the study demonstrated significantly higher scores on the post-test than on the pre-test. 

This finding supports the value-added for all students in Business programs at these colleges. 
Students in traditional programs are not likely to learn significantly more than non-traditional 
students are, according to the ETS assessments. 
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3) There is no significant difference between the mean scores of traditional and non-traditional 
students compared to the national mean . 

This hypothesis is not rejected. Comparing the mean scores of all students in this study to 
the national mean for this test shows no significant difference (t = .649, p = .517). This finding 
varies somewhat when the data are disaggregated. 

Comparing mean scores of traditional students and non-traditional students independently 
with the national mean shows that traditional students scored significantly lower than the national 
mean (t = -2.79, p = .008). In contrast, non-traditional students scored significantly higher (t = 2.40, 

p = 018). 

Further analysis shows that this result also varies by college. Non-traditional students at two 
schools scored significantly higher than the national mean while traditional students at these schools 
did significantly worse. This is consistent with the result in the discussion of in hypothesis 1), 
above, and is consistent with the observation that some students, particularly traditional students, 
did not take the test seriously. Another possible explanation is that the students in the non- 
traditional programs are older, with more Business experience, which translates into their better 
performance on this test. This, of course, is the underlying philosophy of non-traditional programs. 

4) There is no significant correlation between the pre-and post-test scores and the grade point 
averages corresponding to the time the test was taken . 

This hypothesis is rejected at the post-test level, but is not rejected at the pre-test. There is a 
significant positive correlation between post-test scores and GPAs when correlating for all students. 
The correlation between the pre-test and concurrent GPA is not significant. This result is not 
consistent when results are disaggregated by college. This result is unexplained, and suggests the 
finding regarding the pre-test/GPA correlation is not robust. 
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Summary: 

The results are heartening for practitioners in non-traditional programs. These findings 
suggest that students do at least as well, arguably better, than students in traditional programs. Two 
possible explanations are a) that some students, particularly traditional age, did not take the test 
seriously, and b) students in the non-traditional programs are more mature and experienced, and 
bring that experience with them into the classroom. This latter is the underlying premise of these 
programs. 
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